1. Field of the Invention
The present invention relates to the field of data entry in computer systems.
2. Background Art
A typical computer system consists of a central processing unit (CPU), main memory such as random access memory (RAM), a data entry device, including a positioning device, a mass storage device such as one or more disk drives, a display and/or a printer. In the prior art, the data entry device often consists of a keyboard, on which a user enters data by typing. The positioning device of a prior art computer system may consist of a "mouse" or other cursor positioning device.
Computer systems have also been developed that are directed to handwritten data entry rather than keyboard data entry. These systems are often characterized by the use of a pen, stylus, or other writing device, to enter handwritten data directly on the display of the computer system. Alternatively, these systems may provide for a user to enter data on a digitizing tablet or other input device, with the image of the written input displayed on a separate computer display output device. The writing device for entering handwritten or freestyle stroke input information is not limited to a pen or stylus, but may be any input device such as a mouse, trackball, pointer, or even a person's fingers. Such systems are not necessarily limited to receiving data generated by human users; for example, machine generated data may also be inputted and accepted to such systems.
One class of this handwriting entry computer system that receives handwritten data input is referred to as a "pen based" computer system. In a pen based computer system, a writer can input information on a display by "writing" directly on the display. A writing device, such as a pen or stylus, is used to enter information on the display. In a typical pen-based computer system, a user touches the stylus to the display and writes as he would on a piece of paper, by making a series of pen strokes to form letters and words. A line appears on the display that follows the path of travel of the pen point, so that the pen strokes appear on the display as ink would appear on a handwritten page. Thus, the user can enter information into the computer by writing on the display. These strokes are referred to herein as "ink". A related series of strokes or characters are referred to as a "block of ink".
In addition to handwriting, special functions based on input strokes are supported on pen based computer systems. For example, a writer may strike certain previously entered handwritten input by making the strokes of an "X" over the entry. These special functional strokes or actions are referred to herein as "gestures". Pen based computers typically have a display surface that serves as both an input receiving device and as an output display device.
The display of a pen-based entry or other handwriting entry computer system may have bordered regions called "fields" that define locations or areas where data is to be entered, or that are associated with particular actions. For example, if the software application provides a data entry display that is an information form with fields for "First Name", "Last Name", "Age", "Occupation", etc., certain information is generally associated with each field. These locations are referred to here as "objects" or "fields". Each object has a boundary associated with it. When data is entered, the computer must determine which object is to be associated with the writing or gesture. The object intended to be associated with the data input is called the target object. The process of identifying the target object is referred to here as targeting. The method and apparatus that performs the function of targeting is referred to here as a targeting system.
One characteristic of handwriting entry computer systems is the ability to translate original handwritten strokes of ink or blocks of ink into machine readable words or characters for display. This translation is accomplished via a "character recognition" algorithm. The strokes of ink that form handwritten words are translated into, for example, ASCII characters. After the translation, the appearance of the displayed characters is as if they had been typed in via a keyboard.
To translate a handwritten character into a machine readable character, the handwritten character is compared to a library of characters to determine if there is a match. A description, or "template" for each character is defined and stored in memory. Handwritten characters are compared to the stored templates. Match coefficients, reflecting how closely a handwritten character matches the template of a stored character, are calculated for each template character. The template character with the highest match coefficient is identified. The character represented by this template provides the "best fit" for the handwritten character. If the match coefficient for the "best fit" character exceeds a predetermined minimum threshold., the "best fit" character is adopted. If the match coefficient for the "best fit" character is less than the minimum threshold value, no translation is done. If the handwritten character cannot be translated, the character must be re-entered.
A disadvantage of current character recognition algorithms is limited accuracy. Often, handwritten characters are not translated at all or are mistranslated as an ASCII character other than the handwritten character. The mistranslated character must then be rewritten by the user, sometimes repeatedly, until a correct translation is made.
A number of prior art handwriting recognition and translation schemes have been described in United States patents. These prior art schemes are described below.
Fukunaga, U.S. Pat. No. 4,641,354, is directed to an apparatus for recognizing and displaying handwritten characters and figures. The apparatus includes a stylus, a liquid crystal display, and handwritten character and figure recognition means, based upon input stroke information. The Fukunaga system displays the result of recognition as it is completed, erasing from the display screen the stroke information used for recognition, and retaining on the display the stroke information which has not yet been used for recognition. Fukunaga discusses the use of a character recognition algorithm that converts handwritten text as soon as the algorithm determines that the stroke information for a character is completed. To conserve memory, all handwritten information is erased from the display as soon as it is recognized.
Conoval, U.S. Pat. No. 4,703,511, is directed to a handwriting input apparatus and method of determining handwriting dynamics information. The Conoval system uses a portable unit and a code generating stylus that modulates handwriting markings, thereby embedding time code information within the writing path. The coded markings are delivered to the writing surface via electromechanical, electrothermal, or electrostatic printhead devices. After the writing is produced, an image detection means may convert the written image to an electrical signal representation thereof for further dynamic analysis. The writing device of Conoval emits a modulated signal which is memorialized on the writing surface as the user hand writes text. Temporal information is stored in the strokes comprising the handwritten characters. The handwritten characters are displayed as they are written. However, the characters are not line-segmented. Conoval suggests that the writing surface, on which the characters are written, can later be provided to a character recognition algorithm for conversion to typewritten fonts.
The system described in More, U.S. Pat. No. 4,839,634, is a portable electro-optic data input/output, storage, processing, and display device responsive to hand printed text and hand drawn graphics. The More system incorporates a character recognition method optimized for automatic and computationally efficient on-line recognition of hand printed text characters. Hand printed text characters are stored in a compact and standard format, like ASCII, for later display, processing or output to external equipment. In one preferred embodiment, the user places hand printed characters into a text conversion area, where they are recognized, converted to typewritten fonts on a line-by-line basis, and displayed in a document viewing/processing area. With the device of More, a user hand writes a line of characters in a text conversion area. The characters are displayed in this area as they are written. When the user activates a "CONVERT" button on the device, the entire line of handwritten text is operated on by a character recognition algorithm. Any handwritten characters in the text conversion area are erased. The recognition output is displayed as typewritten fonts in a larger, separate document viewing/word processing area. All editing functions are performed on the typewritten fonts in a display area which is separate from the area where data is inputted. Even though character recognition does not take place until the user initiates the recognition algorithm, the user can enter only a single line of unconverted text at a time. More does not allow the user to work with and perform functions on the untranslated handwritten characters. Further, the user is prompted to correct any translating errors as each line is converted. More requires untranslated, handwritten characters to be resident in a specific and limited region.
A portable keyboardless entry computer system is described in Sklarew, U.S. Pat. No. 4,972,496. The invention in Sklarew temporarily displays the hand printed characters and then translates them into pre-selected fonts by way of a character recognition algorithm. The user is prompted to make any corrections to handwritten data as it is being inputted. The system in Sklarew also includes a word processing editor. The portable unit described in Sklarew also suffers from some of the disadvantages of Fukunaga. The device of Sklarew concurrently displays handwritten text and the result of its recognition algorithm on the screen. The user is prompted to correct any errors before the translated characters are permanently transferred to the screen.
Shojima U.S. Pat. No. 4,718,103, and Jeanty, U.S. Pat. No. 5,001,765, are both specifically related to character recognition algorithms.
A disadvantage of the prior art translation systems is that the character recognition and translation functions are invoked in real time, immediately after entry of a handwritten word. When the characters are not translated correctly, the user is prompted to correct the translation or reenter the handwritten data. This interrupts the flow of data entry, increasing data entry time, frustrating the user, and decreasing the usability of handwriting entry computer systems.